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Abstract 

The study reported in this paper involved the employment of specially designed learning activities based on the use 
of a Personal Response System (PRS). Instmctors in variety of disciplines are increasingly using personal response 
systems to increase participation, engagement and active learning. Most studies stress the benefits of using a PRS 
when dealing with large groups; in this paper we stress learning benefits for small classes. We demonstrate in a 
context of two very different courses how the PRS use can foster formative interactions in class as well as focusing 
students on the main discipline-specific concepts and skills. We describe the characteristics of these two courses 
along with their educational goals and how it influenced the design of the PRS-based learning activities. We describe 
the formative interactions in class as triggered by the PRS-based questions, and we describe findings regarding 
students' perceptions of the various contributions of the PRS-based learning activities. 

Keywords: Personal response systems, Formative interactions, Instmctional design 

1. Introduction 

Most of the current studies on use of technologies for learning in higher education focus on distance learning. As 
most of the higher education is still conducted mainly in lecture halls, it make sense to invest more efforts on 
studying possible uses of learning technologies in these contexts. This paper deals with such an example, presenting 
experiences along with respective lessons. Personal Response Systems (PRS) is one of several terms used to describe 
technologies for collecting audience responses. Some of the other terms in use are: clickers, polling systems, 
electronic voting systems, and group/ audience/class response systems. Such systems enable participants to instantly 
respond to posed questions. The system can receive participants’ responses and can provide a representation (or 
several ones) of the collected data. PRS are used in a variety of contexts for a variety of purposes. Some examples 
are: voting for decision making, finding characteristics of an audience, market research, etc. In the last decade PRS 
are used in teaching and training to increase participation, engagement and learning (See for example, Beatty & 
Gerace, 2009; Mareno, Bremner, & Emerson, 2010). 

We present the use of the PRS within the framework of formative assessment. We adapt the view that there is more 
leverage to improve teaching through changing aspects of assessment than there is in changing anything else (Gibbs 
& Simpson, 2004). Black and Wiliam (2009), in a paper titled "Developing the theory of formative assessment" use 
the term "formative interactions" that seems to better capture the context of our study. For the long term, formative 
assessment processes can help students take control of their own learning and support them in becoming 
self-regulated learners (Nicol & Macfarlane-Dick, 2006). 

Most papers about the use of PRS for learning deal with pedagogy and/or technology, without explicit consideration 
of the specifics of the subject-matter to be taught (concepts, capabilities, reasoning procedures etc.). The main thesis 
of this article is that the use of a PRS in a course should reflect the main discipline-specific knowledge and skills and 
should be carefully integrated within the course instructional design. PRS-based questions should support formative 
interactions and should convey to the students the specific learning goals of the course on hand. 

In this paper we describe an exploratory study with emphasize on the formative interactions in class as triggered by 
the PRS-based questions. The second section of this paper includes a description of related work dealing with the use 
of PRS for learning. The third section describes the study goals, setting and tools. The fourth section of this paper 
describes examples of our use of a PRS in two very different courses with very different audiences. The fifth section 
presents findings regarding students' perceptions on the contribution of the PRS-based learning activities. We 
conclude with a summary and discussion. 
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2. Related work - Use of PRS for learning 

Personal Response Systems have potential benefits for both teachers and students (Simpson & Oliver, 2007). For the 
teachers, they provide immediate feedback about the student learning process and allow the teacher to gauge the 
overall comprehension of the concepts involved in the material. For the students, it can be effective for engagement 
in class activities, for promoting interactions among students, for providing immediate feedback, and for facilitating 
the active participation of students in the learning process by discussing the answers given to the questions. 

Even though researchers raise the questions whether it is a ’’teaching innovation or merely an amusing novelty” 
(Lantz, 2010) and whether it is "empowering or compelling reluctant participators” (Graham, Tripp, Seawright, & 
Joeckel, 2007); still most research about the use of PRS in class deal with the empowering potential. For example, 
studies deal with supporting critical thinking (Molbom & Hoekstra, 2010), with promoting the acquisition of 
advanced reasoning skills (DeBourge, 2008), with facilitating the development of problem-solving skills (Levesque, 
2011), and with influencing metacognition (Brady, Seli & Rosenthal, 2013). This empowering use spans a variety of 
subject matters in higher education like Biology (Crossgrove & Curran, 2008), Computer Science (Fan & van Blink, 
2006), and Sociology (Molborn & Hoekstra, 2010). Even though papers deal with PRS use in a variety of courses, 
the considerations of the specific subject-matter are not explicitly dealt with. 

Researchers attribute the success of PRS to the fact that it provides a context for active learning and that questioning 
provokes deeper cognitive processing in the learner, which in turn results in better learning (Campbell & Mayer, 
2008). The immediate feedback can serve as a motivating factor in focusing student attention and allowing 
opportunities for review and reflection (Gauci, Dantas, Williams, & Kemm, 2009). Martyn (2007) mentions two 
important key features that distinguish the use of PRS from active learning that employs questions and discussion in 
class. PRS use provides a mechanism for students to participate anonymously; and it integrates a "game approach” 
that has the potential to engage the students even more than traditional class discussion. The latter fits well with the 
current research regarding the benefits of gamification (Dominguez, 2013). 

Blasco-Arcas, Buil, Hemandez-Ortega, and Sese (2013) developed a conceptual framework, based on quantitative 
analysis, to explain the positive effects and benefits of clickers in enhancing student learning performance. They 
suggest that student perceptions of the interactivity with peers and with the teacher that result from using clickers 
promote active collaborative learning and engagement, which ultimately leads to enhanced learning performance. 

What should be added to this framework is the elements of instructional design that address the design of the 
questions along with the timing for presenting them. Any evaluation, whether formative or summative, forwards a 
message (usually implicitly) to the students what are the main issues of the topic on hand. The use of the PRS-based 
activities has the potential to focus the attention of the students to the main concepts and skills. It is important to 
design the PRS-based activities so that they are spread in a way that enables this focusing goal; and also that the PRS 
is not overused because it might lose its potential for engagement. In this paper we try to demonstrate how the 
specifics of the course should be incorporated into the employment of the PRS-based learning activities to enhance 
the above processes for better learning of the given subject-matter. 

SMS-HIT (Kohen-Vax, Ronen & Bar-Ness, 2012) is a personal response system based on mobile devices for SMS 
and web response provision. The system is designed for teaching purposes enabling instructors to prepare and enact 
personal response activities in actual instructional setting for any subject domain. Readymade activities are stored in 
a repository and could be later on copied, modified and reused. The various types of responses along with the 
respective processing and presentation, provide opportunities to adapt the activity to the instructional context as 
characterized by the course, the topic (and a specific sub-topic), stage in the course, students (number, previous 
knowledge and other characteristics), pedagogical strategy, and more. The SMS-HIT system was used in our study 
for several types of in-class activities, as will be described in the next section. 

3. The Study - Goals, setting and tools 

The study is mainly an exploratory study intended to explore if and how PRS-based activities enhance learning. The 
goal was to Investigate whether the PRS-based activities keep the focus on the learning process instead of on the new 
introduced technology, and moreover, if the PRS-based activities support focusing on the main concepts and the 
main cognitive processes of the specific course. 

The PRS-based activities were designed for two very different courses for very different audiences. One course is 
"Decision Analysis", which is a mandatory course for the Management Information Systems department in our 
college. Courses dealing with decision analysis can have different flavors. Some courses emphasize a descriptive 
view of human decision making behavior, while others emphasize mathematical modeling. Our course deals mainly 
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with quantitative methods for decision analysis under uncertainty, which entails the employment of probability 
theory. The course presents also some psychological aspects of probabilistic thinking and decision making in order to 
make students aware of respective biases. As such a course tends to be technical and involves mathematical 
procedures, it seemed important to keep the focus on the motivation for decision analysis - choose between 
alternatives, use rational tools, and try to overcome known biases. The course lasted 14 weeks, and 34 students 
participated in this course during the semester when the PRS was used. The PRS was used only in part of the 
sessions, when the specially designed activities were appropriate. 

The second course is "Design of computerized games and interactive stories", which is an elective course for the 
behavioral sciences students in our college (Behavioral Science, Education, Psychology, and Sociology). The 
rationale for the course design was to provide a motivating and engaging context for introducing computer 
programming to these students, to empower the students as computer users and to support the development of 
respective higher order cognitive skills such as planning, problem-solving etc. (Or-Bach, 2013). The programming 
environment chosen for this course is Scratch, which is a visual programming environment that lets users create 
interactive, media-rich projects (Maloney et al. 2010; Resnick et al., 2009). Studies of novice programmers (e.g. 
Lister, Simon, Thompson, Whalley, & Prasad, 2006) foster the importance of tracing and “explain in plain English” 
tasks for developing writing/generation of code capabilities. Therefore the PRS-based learning activities that were 
designed for this course emphasized the skills of tracing code to predict its behavior, and of explaining the overall 
goal or meaning of a given code. During the semester when this study was conducted the course lasted 14 weeks 
with a 2 hours session each week in a computer lab and 23 students participated in the course. The SMS-HIT PRS 
was used in most of the sessions, sometimes more than once but with an attempt not to use it too often. 

In both courses the PRS-based questions were presented through the Learning Management System (LMS) that is 
regularly used in our college. The exercises included a link to a site where the response can be written or selected. 

The research tools included: collected data from the PRS use (as collected by the software), instructor’s impressions 
from the class activities that followed the questions’ presentation, and surveys for obtaining students’ perceptions on 
the various contributions of using PRS to enhance learning. A survey was administered to the students in the "Design 
of computerized games and interactive stories" course. The survey was administered at the end of the course, and its 
goal was to investigate the contribution of the specific PRS-based exercises to the learning of Scratch programming 
and to find out what are the aspects where the use of the PRS contributed the most. The survey included 10 
Likert-type items and a free text question regarding the special in-class exercises. An additional item dealt with the 
student’s subjective evaluation of his/her relative mastery of Scratch programming. It seemed important to check this 
because over-confidence or under-confidence might give a different interpretation for the survey results. In the 
Decision analysis course the PRS was used also for collecting students’ attitudes through two questions. One 
question dealt with the contribution to the student’s engagement level and the other question dealt with the 
contribution of the PRS-based learning activities to the learning of the course main concepts. 

4. Examples of PRS-based learning activities with respective findings 

4.1 Introduction - The role of the examples 

The examples of questions in this section are of two types that the specific PRS software (SMS-HIT) supports: open 
- short free text answer and multiple choice one. For each course we present one example for each type. The 
software also provides several representations for the responses and their frequencies (or relative frequencies). For 
both types of questions the results were presented to the students by a visual representation. For the first type - A 
word cloud, and for the second type a column diagram. The general instructional process that was triggered by these 
examples is: time for answering the question, time for reflection after the presentation of the results (answers and 
their relative frequency), time for rechecking answers (when applicable); and time for class discussion on what is 
correct, alternative answers, possible sources for errors and respective misconceptions. 

For each question we present the question, the results from administering the question in class, the following 
instructional process and a short discussion of the implications. 

4.2 The "Decision analysis" course 

Two examples are presented here from the course on decision analysis and decision making. The two questions were 
designed to show potential problems with human probabilistic thinking that can affect decision making. One question 
related to subjective probabilities and the other to known biases. 
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4.2.1 Example 1 

"This event will almost certainly occur. What percentage will you assign to the event occurrence?" 

For such free text answers the SMS-HIT provides a visual representation for the given answers by a word cloud 
representation. The size (and color) of the specific text (in this case the percentage) reflects the relative frequency of 
that input. This representation highlights both the range and the relative frequencies. For this specific example the 
range was 51%-99%, which clearly demonstrate how differently people perceive this verbal estimation of "almost 
certainly". 

4.2.2 Example 2 

This example is a classic question used by Tversky and Kahenman (1983) to discuss biases in probabilistic thinking 
(and thus in decision making). 

"Linda is 31 years old, single, outspoken and very bright. She majored in philosophy. As a student, she was deeply 
concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations. 
Which of the following is more probable: 1. Linda is a bank teller; 2. Linda is a bank teller and is active in a feminist 
movement." 

The total number of answers was 31; 17 chose answer no. 1 and 14 chose answer no. 2. The results were presented to 
the students graphically via the bar diagram produced by the software. It was a good context for bringing up the 
respective probability concepts and rules and confronting students with their violation of the probability conjunction 
rule. This confrontation in a course that emphasizes probability calculations for decision making encourages students’ 
reflection processes and adheres nicely with the course learning goals. 

Additional PRS-based questions that were used during this course dealt with decision making under uncertainty, 
where the students had to choose between alternatives after producing the respective decision trees. These questions 
were used for stressing the goal of the decision trees (instead of "sinking" in calculations) and for discussing 
prevalent errors and misconceptions. 

4.3 The "Design of computerized games and interactive stories” course 

4.3.1 Example 1 

This question required "reading" and tracing of code, and providing an explanation of its goal. It involved a script 
(the code in Scratch terminology) that produces a dashed line on the screen by the walking of a sprite that alternates 
repeatedly between using a pen to mark the walking and walking (the same number of steps) without the pen marks. 
The respective script includes several commands of moving/walking and of lowering the pen (to enable drawing) and 
raising it up. These commands are placed within a programing construct of repetition. The expected description of 
"dashed line" for the resulted outcome requires the student to understanding the commands, to build a mental model 
of the process and to envision the result as an abstracted artifact. All these skills are necessary for being able to 
"write" programs and manage the debugging processes that are part of this activity. 

In order to trigger formative interactions, the list of all the responses was shown to the students and the students were 
asked to identify responses that are equivalent. This brought up for discussion the issues of sematic equivalence and 
abstraction levels. Eight out of 15 responses were "dashed line" or a synonym adjective for the line. Four responses 
were explanations of all the commands without grasping the overall resulted line. For the other responses given to 
this question students were asked to try to figure out the source or reason for the answer that was apparently incorrect. 
One response was "a building", maybe indicating understanding that something will be drawn on the screen but 
without the ability to systematically trace the process. Two responses concentrated on the sprite ("runs to the right", 
"runs quickly"), maybe without understanding the pen commands. 

These formative interactions seemed beneficial for making students more aware of the goals and processes involved 
in the specific task as well as in programming tasks in general. 

4.3.2 Example 2 

This question consisted of two scripts that look very similar and the student is asked whether or not they are 
equivalent (act the same way under all relevant conditions). In fact the two scripts are not equivalent as one of the 
shared commands is in one script within the conditional loop and in the other script outside of that scope. 
Discovering this difference requires a systematic and careful tracing of both scripts under the specific condition 
appearing in both scripts. 

Before any discussion of what is the correct answer, students were presented with the frequency of the "Yes" and 
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"No” responses in order to re-examine their response through re-reading the code. Only then they were asked to 
actually check the behavior of these two scripts by executing both of them. It seemed that for many students the 
combination of mental tracing along with the actual execution is required for the ultimate conviction. Class 
discussion continued with suggestions to pinpoint the difference between the respective programming constructs and 
their scope. The lesson for the instructor in this case was to further stress the programming construct’s scope issue as 
it is nicely addressed by the visual representation of the command. 

5. Findings regarding students* perceptions on the contribution of the PRS-based learning activities 

The goal of the survey that was administered in the "Design of computerized games and interactive stories" course 
was to investigate the contribution of the specific PRS-based exercises to the learning of Scratch programming and to 
find out what are the aspects where the use of the PRS contributed the most. Two of the ten survey items dealt with 
factors related to the use of PRS in general (motivation and engagement) and the rest dealt with specific factors of 
the PRS use that may contribute to the learning of Scratch programming (requirement of mental processing, 
presentation of different answers, requirement to compare between scripts, discussions based on students' answers 
etc.) The scale for the Likert-type items was designed to obtain a finer resolution on the positive side of the scale 
because a goal of the study was to distinguish between the contributions of the different factors and the instructor's 
impression was that students favored in general the PRS use. 

Twenty (out of 23) students filled out the survey. Figure 1 shows the distribution of the levels of appreciation for the 
different contributions of the PRS use as depicted by the ten items, while table 1 presents the averages and standard 
deviation for these items. 
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Figure 1. distribution of the levels of appreciation for the different contributions 
Table 1. Average and s.d. for the survey items dealing with the various contributions of the PRS use. 


item 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

average 

3.6 

3.8 

3.4 

3.9 

3.95 

3.75 

3.55 

3.7 

3.6 

3.65 

s.d. 

1.00 

1.05 

1.21 

0.97 

1.05 

0.92 

0.90 

1.03 

0.95 

1.08 


Two items dealt with the contribution of the PRS-based exercises to motivation (item no. 1) and to engagement (item 
no. 2), facets that are not related specifically to the course learning goals - learning programming with Scratch. As 
can be seen from figure 1, none of the students chose "Do not agree" for these two items. The rest of the items dealt 
with different aspects of the PRS use for learning to program with Scratch. 

From looking at figure 1 and at table 1, one can see that item no. 5 had both the highest average and the highest 
percentage for the two highest levels of agreement. The phrasing of this item (in Hebrew) was "Mental execution of 
a script helps to deeply understand specific commands". This item actually captures the main goal of these exercises 
and a major goal for learning to program. Along with the above observation, the similar averages and distributions do 
not provide enough evidence for a finer analysis of the contributions of the different elements of the instructional 
strategy to learning Scratch programming (e.g. the mental processing, the reflection processes after seeing different 
answers or the follow-up class discussion). Further analysis showed that while the standard deviation for an item was 
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around 1, the standard deviation for a student was only around 0.5. The results might indicate that the aspects are 
highly correlated and students appreciate the combination, and/or that students can not differentiate between the 
aspects and thus formulate a general impression. 

In the Decision analysis course two questions were presented to the students through the PRS. One question dealt 
with the contribution to the student’s engagement level and the other question dealt with the contribution of the 
PRS-based learning activities to the learning of the course main ideas. The questions were of the multiple-choice 
type and table 2 presents the percentage of choices for each of the three options for each question. 


Table 2. the percentage of choices for each of the three options for each question. 


Question goal 

Contributed 

contributed 

Did not 


significantly 


contribute 

Perception about the contribution to engagement 

50% 

43% 

7% 

Perception about the contribution to learning 

64% 

36% 

0% 

Conclusions and Discussion 


The participation rate in answering the PRS-based questions was satisfactory. It was about 80% in the Decision 
Analysis course, and about 70% in the Design of computerized games and interactive stories course. This might be 
due to the anonymity effect. The participation is important for enabling an effective learning process that relates to 
the activation of prior knowledge and to possibly evolving conflicts and contradictions that can bring up productive 
learning (Or-Bach & van Joolingen, 2001). For most of the multiple-choice questions, the frequency diagram that 
was produced and presented to the students exhibited a distribution of opinions. Also the free-text short answers 
exhibited a range and variety of answers. This variety provided a context for discussion as well as informed the 
instructor on the different ways of thinking, and of errors and misconceptions. The purpose of the examples in the 
fourth section of this paper was to demonstrate the type of the PRS-based questions that were employed, the 
responses that the students provided, and the instructional process that followed the question presentation in both 
courses with their different characteristics. The impression of the instructor was that students were more attentive 
and that the students’ commitment to their provided answers supported reflection, productive discussion and thus 
deeper learning. The instructor’s impression was that the use of the PRS fostered communication between the 
students and the instructor and especially among the students. Such interactivities contribute to the articulation of 
students’ thinking and thus to more effective discussions (Blasco-Arcas et al., 2013). 

Results from students’ perceptions regarding the contribution of using the PRS-based activities show that students 
appreciated its contribution. In the design of the survey and the feedback questions we tried to extract a finer analysis 
of the different contributions. Even though there were no significant differences between the perceptions of the 
various contributions, in the decision analysis course there was a preference for the contribution to learning versus 
contribution to engagement. It can be interpreted as a greater appreciation for the integration of content and 
pedagogy than pedagogy by itself. The results from the survey, which involved further elaboration of possible 
sources of contribution, showed that the most appreciated facet matches the disciple specific educational goal. 

Most studies stress the benefits of using a PRS when dealing with large groups; in this paper we stress learning 
benefits even for small classes. We claim that in order to achieve meaningful contribution to learning it is important 
to adapt the use of the PRS, and especially the design of the questions, to the special characteristics and educational 
goals of the course. Black and Wiliam (2009), in their comprehensive paper titled ’’Developing the theory of 
formative assessment”, stress the role of the instructors to engineer into the learning environment the essence of the 
discipline so that the students can eventually operate as effective learners in the discipline. We demonstrated this 
approach through the implementation of PRS use in two very different courses. In the "Decision Analysis" course for 
the students from the department of Management Information Systems, the PRS-based activities were designed to 
stress the issues of modeling for choosing between alternatives, use of rational methods, and efforts to overcome 
known biases. In the "Design of computerized games and interactive stories" course for the Behavioral Sciences 
students, the PRS-based activities were designed to stress the skill of mental tracing of code and the concept of 
abstraction, which are instrumental of programming. 
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